{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "StanfordNLP-101.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "56LiYCkPM7V_",
        "colab_type": "text"
      },
      "source": [
        "# Welcome to StanfordNLP!\n",
        "\n",
        "![Latest Version](https://img.shields.io/pypi/v/stanfordnlp.svg?colorB=bc4545)\n",
        "![Python Versions](https://img.shields.io/pypi/pyversions/stanfordnlp.svg?colorB=bc4545)\n",
        "\n",
        "StanfordNLP is a Python NLP toolkit for core sentence analysis tasks like tokenization, lemmatization, part-of-speech tagging, and dependency parsing. It is built with highly accurate neural network components that enable efficient training and evaluation with your own annotated data, and it comes with pretrained models built on over 70+ [UD](https://universaldependencies.org/) treebanks that support 50+ human languages. Additionally, StanfordNLP provides a stable, officially maintained Python interface to the Java [Stanford CoreNLP](https://stanfordnlp.github.io/CoreNLP/) toolkit.\n",
        "\n",
        "In this tutorial, we will demonstrate how to set up StanfordNLP and annotate text with its native neural network NLP models. For the use of the Python CoreNLP interface, please see other tutorials."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yQff4Di5Nnq0",
        "colab_type": "text"
      },
      "source": [
        "## 1. Installing StanfordNLP\n",
        "\n",
        "StanfordNLP only supports Python 3.6 and above and uses [the PyTorch library](https://pytorch.org/). Installing and importing StanfordNLP are as simple as running the following commands:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "owSj1UtdEvSU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Install; note that the prefix \"!\" is not needed if you are running in a terminal\n",
        "!pip install stanfordnlp\n",
        "\n",
        "# Import the package\n",
        "import stanfordnlp"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4ixllwEKeCJg",
        "colab_type": "text"
      },
      "source": [
        "### More Information\n",
        "\n",
        "For common troubleshooting, please visit our [troubleshooting page](https://stanfordnlp.github.io/stanfordnlp/installation_usage.html#troubleshooting)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aeyPs5ARO79d",
        "colab_type": "text"
      },
      "source": [
        "## 2. Downloading Models\n",
        "\n",
        "You can download models with the `stanfordnlp.download` command. The language can be specified with either a full language name (e.g., \"english\"), or a short ISO 639 code (e.g., \"en\"). \n",
        "\n",
        "By default, models will be saved to your `~/stanfordnlp_resources` directory. If you want to specify your own path to save the model files, you can pass a `resource_dir=your_path` argument.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HDwRm-KXGcYo",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Download an English model into the default directory\n",
        "# Note that if force=True is not specified, the command will be followed with prompts asking for your confirmation.\n",
        "stanfordnlp.download('en', force=True)\n",
        "\n",
        "# Similarly, download a (traditional) Chinese model\n",
        "stanfordnlp.download('zh', force=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7HCfQ0SfdmsU",
        "colab_type": "text"
      },
      "source": [
        "### More Information\n",
        "\n",
        "Pretrained models are provided for 50+ different languages. For all languages and the corresponding short language codes, please check out the [models page](https://stanfordnlp.github.io/stanfordnlp/models.html#human-languages-supported-by-stanfordnlp).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b3-WZJrzWD2o",
        "colab_type": "text"
      },
      "source": [
        "## 3. Processing Text\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XrnKl2m3fq2f",
        "colab_type": "text"
      },
      "source": [
        "### Constructing Pipeline\n",
        "\n",
        "To process a piece of text, you'll need to first construct a `Pipeline` with different `Processor` units. The pipeline is language-specific, so again you'll need to first specify the language (see examples).\n",
        "\n",
        "- By default, the pipeline will include all processors, including tokenization, multi-word token expansion, part-of-speech tagging, lemmatization and dependency parsing. However, you can always specify what processors you want to include with the `processors` argument.\n",
        "\n",
        "- Note that the backend device has to be specified when building the pipeline. By default, the pipeline will always use a CUDA-compatible GPU if it exists on your device to accelerate computing, and it'll fall back to CPU if no such GPU can be found. You can force the pipeline to use CPU regardless by setting `use_gpu=False`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HbiTSBDPG53o",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Build an English pipeline, with all processors by default\n",
        "en_nlp = stanfordnlp.Pipeline(lang='en')\n",
        "\n",
        "# Build a Chinese pipeline, with customized processor list, and force it to use CPU\n",
        "zh_nlp = stanfordnlp.Pipeline(lang='zh', processors='tokenize,lemma,pos,depparse', use_gpu=False)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Go123Bx8e1wt",
        "colab_type": "text"
      },
      "source": [
        "### Annotating Text\n",
        "\n",
        "After a pipeline is successfully constructed, you can get annotations of a piece of text simply by passing the string into the pipeline object. The pipeline will return a `Document` object, which can be used to access detailed annotations from. For example:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k_p0h1UTHDMm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Processing English text\n",
        "en_doc = en_nlp(\"Barack Obama was born in Hawaii.  He was elected president in 2008.\")\n",
        "print(type(en_doc))\n",
        "\n",
        "# Processing Chinese text\n",
        "zh_doc = zh_nlp(\"達沃斯世界經濟論壇是每年全球政商界領袖聚在一起的年度盛事。\")\n",
        "print(type(zh_doc))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DavwCP9egzNZ",
        "colab_type": "text"
      },
      "source": [
        "### More Information\n",
        "\n",
        "For more information on how to construct a pipeline and information on different processors, please visit our [pipeline page](https://stanfordnlp.github.io/stanfordnlp/pipeline.html)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O_PYLEGziQWR",
        "colab_type": "text"
      },
      "source": [
        "## 4. Accessing Annotations\n",
        "\n",
        "Annotations can be accessed from the returned `Document` object. \n",
        "\n",
        "A `Document` contains a list of `Sentence`s, and a `Sentence` contains a list of `Token`s and `Word`s. For the most part `Token`s and `Word`s overlap, but some tokens can be divided into mutiple words, for instance the French token `aux` is divided into the words `à` and `les`. Note that dependency parses are derived over `Word`s.\n",
        "\n",
        "The following example iterates over all English sentences and words, and prints the word information one by one:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B5691SpFHFZ6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "for i, sent in enumerate(en_doc.sentences):\n",
        "    print(\"[Sentence {}]\".format(i+1))\n",
        "    for word in sent.words:\n",
        "        print(\"{:12s}\\t{:12s}\\t{:6s}\\t{:d}\\t{:12s}\".format(\\\n",
        "              word.text, word.lemma, word.pos, word.governor, word.dependency_relation))\n",
        "    print(\"\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ql1SZlZOnMLo",
        "colab_type": "text"
      },
      "source": [
        "Similarly for the Chinese text:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XsVcEO9tHKPG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "for i, sent in enumerate(zh_doc.sentences):\n",
        "    print(\"[Sentence {}]\".format(i+1))\n",
        "    for word in sent.words:\n",
        "        print(\"{:12s}\\t{:12s}\\t{:6s}\\t{:d}\\t{:12s}\".format(\\\n",
        "              word.text, word.lemma, word.pos, word.governor, word.dependency_relation))\n",
        "    print(\"\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dUhWAs8pnnHT",
        "colab_type": "text"
      },
      "source": [
        "Alternatively, you can directly print a `Word` object to view all its annotations:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6_UafNb7HHIg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "word = en_doc.sentences[0].words[0]\n",
        "print(word)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TAQlOsuRoq2V",
        "colab_type": "text"
      },
      "source": [
        "### More Information\n",
        "\n",
        "For all information on different data objects, please visit our [data objects page](https://stanfordnlp.github.io/stanfordnlp/data_objects.html)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hiiWHxYPpmhd",
        "colab_type": "text"
      },
      "source": [
        "## 5. Other Resources\n",
        "\n",
        "- [StanfordNLP Homepage](https://stanfordnlp.github.io/stanfordnlp/index.html)\n",
        "- [FAQs](https://stanfordnlp.github.io/stanfordnlp/faq.html)\n",
        "- [GitHub Repo](https://github.com/stanfordnlp/stanfordnlp)\n",
        "- [Reporting Issues](https://github.com/stanfordnlp/stanfordnlp/issues)\n",
        "- [System Paper](https://nlp.stanford.edu/pubs/qi2018universal.pdf)\n"
      ]
    }
  ]
}